{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "custom_training.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.5"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5rmpybwysXGV"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "colab_type": "code",
        "id": "m8y3rGtQsYP2",
        "colab": {}
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hrXv0rU9sIma"
      },
      "source": [
        "# 定制化训练：基础"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7S0BwJ_8sLu7"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://tensorflow.google.cn/tutorials/customization/custom_training\"><img src=\"https://tensorflow.google.cn/images/tf_logo_32px.png\" />在 Tensorflow.google.cn 上查看</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/zh-cn/tutorials/customization/custom_training.ipynb\"><img src=\"https://tensorflow.google.cn/images/colab_logo_32px.png\" />在 Google Colab 运行</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/zh-cn/tutorials/customization/custom_training.ipynb\"><img src=\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\" />在 Github 上查看源代码</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/zh-cn/tutorials/customization/custom_training.ipynb\"><img src=\"https://tensorflow.google.cn/images/download_logo_32px.png\" />下载此 notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "k2o3TTG4TFpt"
      },
      "source": [
        "在之前的教程中，我们讨论了自动差分，一个基本的机器学习模块。在这个教程中，我们将使用在之前介绍的 Tensorflow 基础语句实现简单的机器学习模型。\n",
        "\n",
        "Tensorflow 也提供了高级神经网络 API（`tf.keras`），可以精简范例代码。我们强烈建议在神经网络方面的工作使用高级的 API。在这篇教程中，我们使用基本规则来训练神经网络，为以后打下牢固基础。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3LXMVuV0VhDr"
      },
      "source": [
        "## 创建"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "NiolgWMPgpwI",
        "colab": {}
      },
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "\n",
        "try:\n",
        "  # Colab only\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "    pass\n",
        "import tensorflow as tf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "eMAWbDJFVmMk"
      },
      "source": [
        "## 变量\n",
        "\n",
        "Tensorflow 中的 tensor 是不可变无状态对象。机器学习模型需要可改变状态，比如模型训练和模型预测的代码是相同的，但变量值随着时间而不同（希望尽量小的 loss）。为了应对随着计算而改变的状态，可以利用 Python 的状态可变性。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "VkJwtLS_Jbn8",
        "colab": {}
      },
      "source": [
        "# 使用 python 状态\n",
        "x = tf.zeros([10, 10])\n",
        "# 等价于 x = x + 2, 不改变原本 x 的值\n",
        "x += 2  \n",
        "print(x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wfneTXy7JcUz"
      },
      "source": [
        "TensorFlow，拥有内建可变状态操作，比使用底层 Python 状态表示更常见的。比如，表示模型的权重，使用 TensorFlow 变量更方便高效。\n",
        "\n",
        "变量是一个对象，这个对象存储着数值，当在 TensorFlow 计算中使用时，会隐式地读取这个存储的数值。有一些操作（`tf.assign_sub`, `tf.scatter_update` 等）会复制 TensorFlow 变量存储的数值。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "itxmrMil6DQi",
        "colab": {}
      },
      "source": [
        "v = tf.Variable(1.0)\n",
        "assert v.numpy() == 1.0\n",
        "\n",
        "# 重新赋值\n",
        "v.assign(3.0)\n",
        "assert v.numpy() == 3.0\n",
        "\n",
        "# 在 TensorFlow 操作中使用 `v`，比如  tf.square() 和重新赋值\n",
        "v.assign(tf.square(v))\n",
        "assert v.numpy() == 9.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-paSaeq1JzwC"
      },
      "source": [
        "当计算梯度时，会自动跟踪使用变量的计算过程。用变量来表示向量时，TensorFlow 会默认使用稀疏更新，这样可以带来计算和存储高效性。\n",
        "\n",
        "使用变量也是一种更快的提醒方式，就是代码的这部分是状态可变的。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BMiFcDzE7Qu3"
      },
      "source": [
        "## 示例：尝试一个线性模型\n",
        "\n",
        "让我们来使用目前为止学到的概念---`Tensor`，`Variable`，和 `GradientTape`---来创建和训练一个简单的模型。一般需要下面这些步骤：\n",
        "\n",
        "1. 定义模型\n",
        "2. 定义损失函数\n",
        "3. 获取训练数据\n",
        "4. 通过训练数据运行模型，使用 \"optimizer\" 来调整变量以满足数据\n",
        "\n",
        "在这个教程中，我们使用一个简单线性模型作为示例：`f(x) = x * W + b`，有2个变量- `W` 和 `b`。另外，我们会生成数据让训练好的模型满足 `W = 3.0` 和 `b = 2.0`。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "gFzH64Jn9PIm"
      },
      "source": [
        "### 定义模型\n",
        "\n",
        "定义一个简单的类封装变量和计算"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "_WRu7Pze7wk8",
        "colab": {}
      },
      "source": [
        "class Model(object):\n",
        "  def __init__(self):\n",
        "    # 初始化变量值为(5.0, 0.0)\n",
        "    # 实际上，这些变量应该初始化为随机值\n",
        "    self.W = tf.Variable(5.0)\n",
        "    self.b = tf.Variable(0.0)\n",
        "\n",
        "  def __call__(self, x):\n",
        "    return self.W * x + self.b\n",
        "\n",
        "model = Model()\n",
        "\n",
        "assert model(3.0).numpy() == 15.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xa6j_yXa-j79"
      },
      "source": [
        "### 定义损失函数\n",
        "\n",
        "损失函数用来衡量在给定输入的情况下，模型的预测输出与实际输出的偏差。我们这里使用标准 L2 损失函数。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Y0ysUFGY924U",
        "colab": {}
      },
      "source": [
        "def loss(predicted_y, desired_y):\n",
        "  return tf.reduce_mean(tf.square(predicted_y - desired_y))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qutT_fkl_CBc"
      },
      "source": [
        "### 获取训练数据\n",
        "\n",
        "我们来生成带噪声的训练数据。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gxPTb-kt_N5m",
        "colab": {}
      },
      "source": [
        "TRUE_W = 3.0\n",
        "TRUE_b = 2.0\n",
        "NUM_EXAMPLES = 1000\n",
        "\n",
        "inputs  = tf.random.normal(shape=[NUM_EXAMPLES])\n",
        "noise   = tf.random.normal(shape=[NUM_EXAMPLES])\n",
        "outputs = inputs * TRUE_W + TRUE_b + noise"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-50nq-wPBsAW"
      },
      "source": [
        "在训练模型之前，我们来看看当前的模型表现。我们绘制模型的预测结果和训练数据，预测结果用红色表示，训练数据用蓝色表示。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "_eb83LtrB4nt",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.scatter(inputs, outputs, c='b')\n",
        "plt.scatter(inputs, model(inputs), c='r')\n",
        "plt.show()\n",
        "\n",
        "print('Current loss: '),\n",
        "print(loss(model(inputs), outputs).numpy())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sSDP-yeq_4jE"
      },
      "source": [
        "### 定义训练循环\n",
        "\n",
        "我们已经定义了网络模型，并且获得了训练数据。现在对模型进行训练，采用[梯度下降](https://en.wikipedia.org/wiki/Gradient_descent)的方式，通过训练数据更新模型的变量（`W` 和 `b`）使得损失量变小。梯度下降中有很多参数，通过 `tf.train.Optimizer` 实现。我们强烈建议使用这些实现方式，但基于通过基本规则创建模型的精神，在这个特别示例中，我们自己实现基本的数学运算。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MBIACgdnA55X",
        "colab": {}
      },
      "source": [
        "def train(model, inputs, outputs, learning_rate):\n",
        "  with tf.GradientTape() as t:\n",
        "    current_loss = loss(model(inputs), outputs)\n",
        "  dW, db = t.gradient(current_loss, [model.W, model.b])\n",
        "  model.W.assign_sub(learning_rate * dW)\n",
        "  model.b.assign_sub(learning_rate * db)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RwWPaJryD2aN"
      },
      "source": [
        "最后，我们对训练数据重复地训练，观察 `W` 和 `b` 是怎么变化的。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "XdfkR223D9dW",
        "colab": {}
      },
      "source": [
        "model = Model()\n",
        "\n",
        "# 收集 W 和 b 的历史数值，用于显示\n",
        "Ws, bs = [], []\n",
        "epochs = range(10)\n",
        "for epoch in epochs:\n",
        "  Ws.append(model.W.numpy())\n",
        "  bs.append(model.b.numpy())\n",
        "  current_loss = loss(model(inputs), outputs)\n",
        "\n",
        "  train(model, inputs, outputs, learning_rate=0.1)\n",
        "  print('Epoch %2d: W=%1.2f b=%1.2f, loss=%2.5f' %\n",
        "        (epoch, Ws[-1], bs[-1], current_loss))\n",
        "\n",
        "# 显示所有\n",
        "plt.plot(epochs, Ws, 'r',\n",
        "         epochs, bs, 'b')\n",
        "plt.plot([TRUE_W] * len(epochs), 'r--',\n",
        "         [TRUE_b] * len(epochs), 'b--')\n",
        "plt.legend(['W', 'b', 'true W', 'true_b'])\n",
        "plt.show()\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "vPnIVuaSJwWz"
      },
      "source": [
        "## 下一步\n",
        "\n",
        "在这个教程中，我们讨论了 `Variable`，而且创建和训练了一个简单的线性模型，使用了在此之前所学习的 TensorFlow 知识点。\n",
        "\n",
        "理论上，掌握了 TensorFlow 这些知识点即可用于机器学习研究。实际上，采用高级的 API 比如 `tf.keras` 是更方便的，特别是神经网络，因为它提供了更高级别的内建模块（命名为 \"layers\"），可以保存和恢复状态，还有配套的损失函数和优化策略等。\n"
      ]
    }
  ]
}
